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Abstract 

Within the K-12 online learning environment there are a variety of standards that designers can utilize 
when creating online courses. To date, the only research-based standards available are proprietary in 
manner. As such, many jurisdictions have begun adopting online course design standards from the 
leading advocacy organization, which that have yet to be validated from a research perspective. This 
article reports on the second phase of a three-stage study designed to examine the validity and reliability 
of the iNACOL National Standards for Quality Online Courses. Phase two utilizes two groups of expert 
reviewers to examine and provide feedback with goal of further refining these standards (after the 
standards had been scrutinized through the lens of the available K-12 online learning literature). 
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Improving the K-12 Online Course Design Review Process: Experts 
Weigh in on iNACOL National Standards for Quality Online Courses 

K-12 online course designers have numerous options when contemplating standards to guide their 
development of asynchronous course content; however, not all standards are freely accessible. Some 
institutions, such as the Virtual High School (VHS), have their own publicly available, in-house process 
(Kozma, Zucker, & Espinoza, 1998), while other institutions use standards that are part of a proprietary 
system used by certified experts, such as the Quality Matters (QM, 2014). In 2007, and then later updated 
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in 2011, the International Association for K-12 Online Learning (iNACOL, 2011) released their National 
Standards for Quality Online Courses. These standards were largely based on standards released earlier 
by the Southern Region Education Board (SREB), with some additions due to iNACOL’s involvement in 
the Partnership for 21 st Century Skills initiative (National Association for K-12 Online Learning [NACOL], 
2007). The iNACOL standards used a rubric that covered five different areas (i.e., content, instructional 
design, student assessment, technology, and course evaluation and support) to review the overall quality 
of a course (iNACOL, 2011). Since its initial release, the standards have been implemented in a variety of 
jurisdictions, including for use in states such as Michigan and Texas (“Making Online Learning 
Accessible,” 2015; Oakland Schools, 2015). However, even as the standards remain popular with 
legislators and policymakers, there has been no research published on the validity of the standards or a 
review as to how they relate specifically to online course design. 

The study reported in this article follows an earlier phase in the validation of the iNACOL standards (see 
Adelstein & Barbour, 2016). Phase one of this larger research initiative reviewed the construct validity of 
the iNACOL standards (Drost, 2011). Using contemporary research, each of the 52 elements found in the 
iNACOL standards were reviewed to determine the level of support each standard had within the research 
literature. Each standard was compared to research into K-12 online learning, as well as the broader field 
of online learning and course design. The following article describes phase two of this validation process, 
which consisted of three rounds of expert review over the revised iNACOL standards from the first phase. 
The authors will begin by briefly discussing the current state of K-12 online course design literature. The 
three phases of the expert review will be outlined, detailing the process and results. Finally, the revised K- 
12 online course design rubric will be discussed. 


Literature Review 

K-12 online learning is not a new concept. Prior to the widespread use of the World Wide Web, students 
and instructors would be able to connect via telephone or correspond through the postal service (Clark, 
2013). As the opportunity for K-12 online learning increased, it should not be surprising that many 
courses were designed using the same principles that designers applied to these legacy distance models, as 
well as to face-to-face courses (Barbour & Adelstein, 2013a; Barbour, Morrison, & Adelstein, 2014). 
Instead of telephones and the postal service, chat rooms and email were utilized (Perrin & Mayhew, 
2000). As websites and learning management systems (LMS) came into existence, courses began to take 
and copy from traditional face-to-face courses (Barbour, 2007). However, it became apparent that there 
were widening differences between the two environments. Effective online educators, for example, had to 
utilize skillsets better suited for K-12 online environments (Davis et al., 2007). As educators had to shift 
their way of thinking, the demand for an overhaul in course design began to form. 

Research, specifically about K-12 online course design, has been limited (Barbour, 2013; Barbour & 
Adelstein, 2013b). There have been studies conducted that focus on specific programs, such as VHS or the 
Florida Virtual Schools (FLVS) (Kozma et al., 1998; Zucker, 2005). In both instances, the design of the 
online course is strongly considered along with other aspects. VHS requires its educators to take a 
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mandatory graduate level course that has a focus on design within the LMS (Zucker & Kozma, 2003). The 
FLVS utilizes a team approach consisting of subject matter experts, project managers, instructional 
designers, and web developers (Johnston, 2004). The team process has proven successful for FLVS, but it 
is a very unique system (Barbour & Reeves, 2009). 

As K-12 online learning has continued to mature and evolve, best practice standards that include aspects 
of course design have also been released (iNACOL, 2011; QM, 2014). Some of these standards are 
proprietary, such as those found in the QM system. Beginning as a 3-year Fund for the Improvement of 
Postsecondary Education grant in 2003 (Legon & Runyon, 2007), the first QM rubric was formed in 
2004. QM gradually became an entire process for online course review (Shattuck, 2007). The current 
rubric utilizes eight general standards (i.e., course overview and introduction, learning objectives, 
assessment and measurement, instructional materials, learner interaction and engagement, course 
technology, learner support, and accessibility), while the program offers to train staff for peer reviews, 
course design, and more (MarylandOnline, 2013). However, even though they have never been tested for 
validity, the iNACOL (2011) standards are an easy place for K-12 online course designers to begin because 
the standards and rubric are publically available and non-proprietary. 


Methodology 

Upon completion of the construct validity phase of this research initiative (see Adelstein & Barbour, 
2016), the next stage was the content validity of the revised rubric. The purpose was to test the design of 
the new rubric through expert review. It was recommended to involve content-area experts, as content 
validity is a result of their verification that the rubric meets the standards as outlined in phase one 
(Roblyer & Wiencke, 2003; Taggart, Phifer, Nixon, & Wood, 2001). Roblyer and Wiencke (2003) denoted 
that a properly designed rubric used in educational technology is a meaningful way to both assess and 
guide practitioners. It should not be surprising to see a leader in the field, such as QM, used a rubric for 
their proprietary design standards during the creation process (Hixon, Barczyk, Buckenmeyer, & 
Feldman, 2011). 

Eight experts, who were divided into two groups, reviewed the standards over the course of three rounds, 
examining each standard from a course design perspective. The experts were selected based on their 
background and experience in K-12 online education (see Table 1). 

Table 1 


Description of the Two Expert Review Groups 


Group A 

Group B 

Ron (all names are pseudonyms) 

• Researcher with approximately 20 years’ 
experience in K-12 online learning. 

Jason 

• Educator with experience in K-12 online 
curriculum and assessment design . 
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Louise 

• Administrator with over 20 years’ 
experience in K-12 online learning. 

Amanda 

• Administrative responsibilities in online 
education for 8 years, 15 years overall in 
education. 

Joanne 

• Educator with over 20 years in both online 
and traditional K-12 and higher education. 

Kim 

• Educator for 16 years, half of which in K- 
12 online learning. 

Connor 

• Educator, administrator, and designer with 

12 years of experience in online education. 

Kelly 

• Educator with five years’ experience in K- 
12 online educational research. 


Specifically, each group consisted of a researcher, administrator, designer, and teacher; all of whom had 
been directly involved with K-12 online learning. 

During round one, each of the experts received a document containing the 52 iNACOL elements listed 
under the five main standards based on the results of the first phase of this research initiative. The 
document was color coded to indicate the nature of research supported for each standard (i.e., green for 
significant K-12 online learning research support, yellow for limited K-12 online learning research 
support, or orange for supported only by non-K-12 literature). There were also two additional sections 
added to the end of the document. The first section offered four new standards that were found to be 
present in the K-12 online learning research, while the second suggested combining elements that were 
seen as similar in scope. In round one, the experts were asked to rate the importance of each standard as 
it related to course design using a basic Likert scale (i.e., 1 for low relevancy, 2 for some relevancy, and 3 
for significant relevancy). An area for comments was also included for each section. 

After compiling the ratings from round one, a second document was created that listed the average rating 
for each of the standards and the comments that experts made. Based upon both the raw rating, as well as 
expert suggestions, the researcher made suggestions about revising or removing certain standards. 
Experts were asked to select one of four options (i.e., keep the standard as is, revise the standard, combine 
with another standard, or delete the standard) and to provide a written rationale for that decision. 

The responses from round two were again compiled in a new document that consisted of three sections: 

1. standards where there were general agreement that should be kept as written, 

2. standards where the expert feedback from the previous two rounds that had a clear consensus for 
either revision or deletion, and 

3. standards that did not have a clear consensus from the experts and would require further 
discussion. 

The experts’ feedback from the previous rounds was listed under each standard. Round three consisted of 
60-minute discussion with each expert group using Google Hangout that focused on the second and third 
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sections of the round three document. 1 During the Google Hangout, the researcher facilitated discussion 
around the standards recommended for revision or deletion until a consensus was achieved on whether to 
revise or delete a particular standard, as well as the specific wording for any revised standards. 


Results 

In this section, we organize the data using the complete iNACOL National Standards for Quality Online 
Courses, broken down by section (see Tables 2-7). The revised iNACOL standards based on the expert 
review are provided in Appendix. 

Table 2 

Section A: Content 


Academic content standards and assessments 

Round one 
average 

Round two responses 

Element At: The goals and objectives clearly state 
what the participants will know or be able to do at the 
end of the course. The goals and objectives are 
measurable in multiple ways 

3 

N/A 

Element A2: The course content and assignments are 
aligned with the state’s content standards, common 
core curriculum, or other accepted content standards 
set for Advanced Placement courses, technology, 
computer science, or other courses whose content is 
not included in the state standards. 

2.875 

N/A 

Element A3: The course content and assignments are 
of sufficient rigor, depth, and breadth to teach the 
standards being addressed. 

2.625 

N/A 

Element A4: Information literacy and communication 
skills are incorporated and taught as an integral part 
of the curriculum. 

2-5 

N/A 

Element A5: Multiple learning resources and 
materials to increase student success are available to 
students before the course begins. 

2.25 

Keep standard as is: 3 

Revise standard: 3 

Combine with another: 0 

Delete standard: 3 
(Revise/ Delete: 1) 

Course overview and introduction 

Round one 
average 

Round two responses 

Element A6: A clear, complete course overview and 
syllabus are included in the course. 

3 

N/A 

Element A7: Course requirements are consistent with 
course goals, are representative of the scope of the 

2.875 

N/A 


1 Due to a last minute emergency situation, one expert from Group A (Connor) was unable to attend the Google 
Hangout. 
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course and are clearly stated. 



Element A8: Information is provided to students, 
parents, and mentors on how to communicate with 
the online instructor and course provider. 

3 

N/A 

Legal and acceptable use policies 

Round one 
average 

Round two responses 

Element A9: The course reflects multi-cultural 
education, and the content is accurate, current, and 
free of bias or advertising. 

2-75 

N/A 

Element A10: Expectations for academic integrity, use 
of copyrighted materials, plagiarism, and netiquette 
(Internet etiquette) regarding lesson activities, 
discussions, and e-mail communications are clearly 
stated. 

2.875 

N/A 

Element Alt: Privacy policies are clearly stated. 

2.5 

N/A 

Instructor resources 

Round one 
average 

Round two responses 

Element A12: Online instructor resources and notes 
are included. 

2-375 

Keep standard as is: 3 

Revise standard: 2 

Combine with another: 2 

Delete standard: 2 
(Delete/Combine: 1) 

Element A13: Assessment and assignment answers 
and explanations are included. 

2-5 

N/A 


Section A was highly regarded by the experts in terms of significance to course design. In the first round, 
there was overwhelming agreement to keep the majority of the elements in some form, with the 
exceptions of A5 (i.e., Multiple learning resources and materials to increase student success are available 
to students before the course begins) and A12 (i.e., Online instructor resources and notes are included). 
Both elements were further discussed in round two, with experts still divided on how to move forward. All 
experts shared in round three that the phrasing of A5 was problematic, questioning how realistic it was to 
have all materials present before the course begins. Ron mentioned that due to the logistics of certain 
courses, having all material available: 

is technically not possible in some settings. Because you do an enrollment and that’s when the 
students are there and they can’t get access to the course until they are enrolled and they are 
enrolled at date of start. So it’s not physically possible. 

It was suggested, and agreed upon by Group A, to change the wording to “All course materials are 
available to students at the course start.” Group B, on the other hand, did not think the element was 
appropriate. Amanda noted: 

you really don’t know what’s going to work until the students start the course and actually, you 
know, get their feedback as to what’s working and what’s not working. The other thing is, is that 
your course materials could be specific to that student as well, like some students may better at, 
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um, a virtual lab or something else, and another student might learn better by watching a video or 
doing something else. 

With other elements in the rubric discussing additional materials, Group B moved to delete A5 (i.e., 
Multiple learning resources and materials to increase student success are available to students before the 
course begins). 

The round two discussions of A12 (i.e., Online instructor resources and notes are included) lead to a 
suggestion of combining it with A13 (i.e., Assessment and assignment answers and explanations are 
included) or keeping it as is. Both groups were quick to lean towards combining the elements. Ron 
suggested a further revision to include the pedagogy behind the material, as this understanding would 
help teachers “to grade [the assignment] appropriately, but they would also be given the grading rubrics 
which they would then communicate clearly in an easy to understand manner to the students and 
parents.” The rest of the Group A members agreed. 

Round one did include two suggestions from experts that were put forth in round two. The first looked to 
combine At (i.e., The goals and objectives clearly state what the participants will know or be able to do at 
the end of the course; the goals and objectives are measurable in multiple ways), A6 (i.e., A clear, 
complete course overview and syllabus are included in the course), and A7 (i.e., Course requirements are 
consistent with course goals, are representative of the scope of the course and are clearly stated) due to 
similarities. Experts were unanimous on combining the elements, with both groups agreeing on the 
suggested wording put forth in round three. The other suggestion was to delete A4 (i.e., Information 
literacy and communication skills are incorporated and taught as an integral part of the curriculum), with 
an expert wondering if it was better suited at a program level and not at the course level. Group A had 
little discussion, as all agreed it was too broad and not a part of the course design. Group B strongly 
thought that the communications piece was already handled in element B9 (i.e., The course design 
includes explicit communication/activities (both before and during the first week of the course) that 
confirms whether students are engaged and are progressing through the course. The instructor will follow 
program guidelines to address non-responsive students.), but perhaps the information literacy should 
remain. For example, Kelly liked the idea that information literacy should be 

embedded in the course design... I really do think that this might be a program related piece 
because it is overarching whole content areas, so it’s not specific to a course design, but it should 
be interwoven into the courses specifically. 

Group B agreed, and revised A4 to read, “Information literacy is incorporated as an integral part of the 
course.” 

Eventually, a decision had to be made regarding the differences between Group A and B results for 
element A4, A5, and A12. This was accomplished by reviewing the current K-12 literature against the 
expert comments from all three rounds. Element A4 was deleted, with the thought that information 
literacy should have a focus at the program or curriculum level, and not in the course design. Element A5 
was kept in the rubric using Group A’s wording. Research showed it was important for the students to 
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have access to the materials before the course begins, allowing them time to make sure everything is 
compatible with personal technology. Finally, A12 and A13 were combined using Group A’s suggestions as 
well. 

Table 3 

Section B: Instructional Design Elements 


Instructional and audience analysis 

Round one 
average 

Round two responses 

Element Bi: Course design reflects a clear 
understanding of all students’ needs and incorporates 
varied ways to learn and master the curriculum. 

2.875 

N/A 

Course, unit and, lesson design 

Round one 
average 

Round two responses 

Element B2: The course is organized by units and 
lessons that fall into a logical sequence. Each unit and 
lesson includes an overview describing objectives, 
activities, assignments, and resources to provide 
multiple learning opportunities for students to master 
the content. 

2.625 

N/A 

Instructional strategies and activities 

Round one 
average 

Round two responses 

Element B3: The course instruction includes activities 
that engage students in active learning. 

3 

N/A 

Element B4: The course and course instructor provide 
students with multiple learning paths, based on 
student needs that engage students in a variety of 
ways. 

2.875 

N/A 

Element B5: The course provides opportunities for 
students to engage in higher-order thinking, critical 
reasoning activities and thinking in increasingly 
complex ways. 

2.875 

N/A 

Element B 6 : The course provides options for the 
instructor to adapt learning activities to accommodate 
students’ needs. 

2.875 

N/A 

Element B7: Readability levels, written language 
assignments, and mathematical requirements are 
appropriate for the course content and grade-level 
expectations. 

2-75 

N/A 

Communication and interaction 

Round one 
average 

Round two responses 

Element B 8 : The course design provides opportunities 
for appropriate instructor-student interaction, 
including opportunities for timely and frequent 
feedback about student progress. 

2.875 

N/A 

Element B9: The course design includes explicit 
communication/activities (both before and during the 

2-375 

Keep standard as is: 2 

Revise standard: 6 
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first week of the course) that confirms whether 
students are engaged and are progressing through the 
course. The instructor will follow program guidelines 
to address non-responsive students. 


Combine with another: 0 

Delete standard: 1 
(Revise/Delete: 1, 

Keep/Revise: 1) 

Element Bio: The course provides opportunities for 
appropriate instructor-student and student-student 
interaction to foster mastery and application of the 
material. 

2.5 

N/A 

Resources and materials 

Round one 
average 

Round two responses 

Element B11: Students have access to resources that 
enrich the course content. 

2-375 

Keep standard as is: 3 

Revise standard: 3 

Combine with another: 0 

Delete standard: 2 


Much like Section A, Section B only had two elements, B9 (i.e., The course design includes explicit 
communication/activities (both before and during the first week of the course) that confirms whether 
students are engaged and are progressing through the course. The instructor will follow program 
guidelines to address non-responsive students.) and B11 (i.e., Students have access to resources that 
enrich the course content), which required further discussion in round two. The wording of B9 was a point 
of contention for a few experts, with the element only mentioning the importance of checking engagement 
before and during the first week. There was a strong overall push to revise the element to include practices 
throughout the course, which lead to the proposed rewording, “The course design includes explicit 
communication/activities at multiple intervals throughout the course that confirms whether students are 
engaged and are progressing through the course. The instructor will follow program guidelines to address 
non-responsive students.” Group B was in agreement with suggestion, while Group A continued the 
discussion. Joanne mentioned that courses also come with tools to assess engagement, and these tools 
should be mentioned and used. She was also concerned with the length of the first sentence, so it was split 
into two sentences for final consideration. 

B11 was debated at length in both expert groups. There were numerous suggestions from round two, such 
as revising to include examples or combining with either A5 (i.e., Multiple learning resources and 
materials to increase student success are available to students before the course begins) or B2 (i.e., The 
course is organized by units and lessons that fall into a logical sequence. Each unit and lesson includes an 
overview describing objectives, activities, assignments, and resources to provide multiple learning 
opportunities for students to master the content.). To start the conversation, the researcher offered the 
suggested revision, “Course design provides students with resources (e.g., alternate assignments, 
multimedia, simulations) that enrich course content”. Group A agreed with the suggestion, with one edit 
recommended from Ron to include mention of Universal Design for Learning (UDL). It was suggested to 
include it both in this standard and at the beginning of the new rubric. 


Group B had already eliminated A5, knowing that B2 and B11 covered much of the same territory. A B2 
revision was previously agreed upon, but the group was quick to see similarities. Jason summed up the 
group’s thoughts when he commented 
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B2 seems to be talking about the overview, and Bn is what is actually there, I guess. Or at least 
describing the opportunities, then. To go along with the overview....It just seems like they need to 
be focused together, to make them one. 

Further, Kelly suggested, “Course design provides students with resources (e.g., alternate assignments, 
multimedia, simulations) that enrich course content. Each unit and lesson includes an overview of the key 
objectives that incorporate a variety of activities, assignments, and resources to provide multiple learning 
opportunities for students to master the content.” The experts in Group A agreed. 

There were two expert suggestions from round one that impacted B2 and Bio (i.e., The course provides 
opportunities for appropriate instructor-student and student-student interaction to foster mastery and 
application of the material). The concern over B2 stemmed from the use of the word logical, which 
appeared to lock the element into a traditional mode of design. Group B was quick to agree upon the 
revision, which simply eliminated “that fall into a logical sequence” from the end of the first sentence. 
Group A shifted their conversation to the use of units and lessons, with Joanne offering up modules. Ron 
agreed, adding 

When we design courses, we design them around weeks. Not units not lessons, but around weeks. 
And I don’t know if units and lessons precludes weeks, but I’m also not sure that it encourages 
that. And units and modules is better. But I would go around, I think organized by modules and 
take out the units. 

Louise was unsure of eliminating units, but came to an understanding that the delivery depends on the 
instructor and mechanisms used. Therefore, modules could stand alone. 

Bio was questioned by an expert for the use of foster, which implied that mastery only comes from the 
suggestions listed in the element. A revision, “The course provides opportunities (e.g., instructor-student 
and student-student interaction, assessments, access to resources) for mastery and application of the 
material,” was suggested to the experts. Group A unanimously agreed, while Kelly had a further revision 
for Group B. Her thought was to keep the examples listed in the element similar to one another by relating 
each interaction to the student. The list was changed to “student-instructor interaction, student-student 
interaction, student-course content, student-LMS,” and experts were content to move on. 

After collecting the expert suggestions, a final decision was made on how to phrase B2, B9, Bio, and B11. 
The most complex of the revisions involved B2 and B11. The similarities brought up by Group B were 
logical, and the reasoning from Jason was enough to move forward with a combination. Group A’s 
suggestion of changing units and lessons to modules was taken under consideration and added to the final 
wording. Group A’s addition of tools and punctuation were accepted for B9, and Group B’s wording was 
used for Bio. 
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Table 4 

Section C: Student Assessment Elements 


Evaluation strategies 

Round one 
average 

Round two responses 

Element Cl: Student evaluation strategies are 
consistent with course goals and objectives, are 
representative of the scope of the course, and are 
clearly stated. 

3 

N/A 

Element C2: The course structure includes adequate 
and appropriate methods and procedures to assess 
students’ mastery of content. 

2-75 

N/A 

Feedback 

Round one 
average 

Round two responses 

Element C3: Ongoing, varied, and frequent 
assessments are conducted throughout the course to 
inform instruction. 

2-375 

Keep standard as is: 2 

Revise standard: 6 

Combine with another: 0 

Delete standard: 0 

Element C4: Assessment strategies and tools make the 
student continuously aware of his/her progress in 
class and mastery of the content. 

2.875 

N/A 

Assessment resources and materials 

Round one 
average 

Round two responses 

Element C5: Assessment materials provide the 
instructor with the flexibility to assess students in a 
variety of ways. 

2.625 

N/A 

Element C6: Grading rubrics are provided to the 
instructor and may be shared with students. 

2.625 

N/A 

Element C7: The grading policy and practices are easy 
to understand. 

2-75 

N/A 


Taken as a whole, Section C was positively viewed by the experts, with only C3 (i.e., Ongoing, varied, and 
frequent assessments are conducted throughout the course to inform instruction) averaging below a 
cumulative 2.5 score. The use of the word “frequent” was an issue for the majority of experts, and it was 
suggested to replace it with “quality.” Both groups were quick to agree with the new wording. 

Three expert suggestions were taken from round one and shared with the group as a whole. The first was 
C2 (i.e., The course structure includes adequate and appropriate methods and procedures to assess 
students’ mastery of content), with the wording “adequate and appropriate” seeming too vague, leaving 
some experts to wonder who determines this. The initial comments from round two were fairly split 
between keeping the original wording and revising the element. Group A promptly decided that the 
original, while a bit vague, gave enough direction for design. Group B, on the other hand, moved to 
eliminate and not replace “adequate and appropriate.” 
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C6 (i.e., Grading rubrics are provided to the instructor and may be shared with students), according to 
one expert, suggested that the word “may” implies the rubric does not need to be shared with students. 
Another expert was concerned that a rubric will be forced upon a qualified teacher. A rubric must be 
supplied in the course, but a qualified instructor should have final say over which rubric to use. While 
there was unanimous agreement amongst the experts that the instructor will share the rubric with 
students, there was some discussion as to the phrasing of the final revision. The suggested wording 
supplied used, “Suggested grading rubrics are provided to the instructor. The instructor will share a final 
grading rubric with students.” Group B accepted the revision, but Group A was concerned over 
misinterpretations about the word “final,” as some might view it in the context of a final exam. Ultimately, 
“final” was replaced by “chosen” in the element. 

Experts were also concerned over language in C7 (i.e., The grading policy and practices are easy to 
understand), and looked to replace “easy to understand” with “clearly communicated.” Group B 
unanimously agreed, while Ron from Group A suggested both phrases should be used. Louise and Ron 
offered continued revisions by adding “to students and parents” at the end, as they are the stakeholders 
who will interpret the policies. 

C2, C6, and C7, had minor revision details that had to be accounted for. C2 was kept as is, as the wording, 
even though vague in nature, gives some direction to the designer. Group A’s version of C6 was kept to 
avoid misinterpretation, and C7 was also finalized by group A. Much like C2, the wording gives 
appropriate direction to the designer. 

Table 5 

Section D: Technology 


Course architecture 

Round one 
average 

Round two responses 

Element Di: The course architecture permits the 
online instructor to add content, activities, and 
assessments to extend learning opportunities. 

2-375 

Keep standard as is: 4 

Revise standard: 4 

Combine with another: 0 

Delete standard: 0 

Element D2: The course accommodates multiple 
school calendars; e.g., block, 4x4, and traditional 
schedules. 

2 

Keep standard as is: 3 

Revise standard: 1 

Combine with another: 0 

Delete standard: 4 

User interface 

Round one 
average 

Round two responses 

Element D3: Clear and consistent navigation is 
present throughout the course. 

2.875 

N/A 

Element D4: Rich media are provided in multiple 
formats for ease of use and access in order to address 
diverse student needs. 

2.714 

N/A 

Technology requirements and 
interoperability 

Round one 
average 

Round two responses 
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Element D5: All technology requirements (including 
hardware, browser, software, etc.) are specified. 

2-75 

N/A 

Element D6: Prerequisite skills in the use of 
technology are identified. 

2-375 

Keep standard as is: 2 

Revise standard: 2 

Combine with another: 1 

Delete standard: 3 

Element D7: The course uses content-specific tools 
and software appropriately. 

2-375 

Keep standard as is: 3 

Revise standard: 2 

Combine with another: 2 

Delete standard: 2 
(Revise/ Delete: 1) 

Element D8: The course is designed to meet 
internationally recognized interoperability standards. 

1-5 

Keep standard as is: 2 

Revise standard: 2 

Combine with another: 0 

Delete standard: 5 
(Keep/Delete = 1) 

Element D9: Copyright and licensing status, including 
permission to share where applicable, is clearly stated 
and easily found. 

2-375 

Keep standard as is: 5 

Revise standard: 2 

Combine with another: 0 

Delete standard: 2 
(Revise/Delete: 1) 

Accessibility 

Round one 
average 

Round two responses 

Element Dio: Course materials and activities are 
designed to provide appropriate access to all students. 
The course, developed with universal design principles 
in mind, conforms to the U.S. Section 504 and Section 
508 provisions for electronic and information 
technology as well as the W3C’s Web Content 
Accessibility Guidelines (WCAG 2.0). 

3 

N/A 

Data security 

Round one 
average 

Round two responses 

Element D11: Student information remains 
confidential, as required by the Family Educational 
Rights and Privacy Act (FERPA). 

3 

N/A 


Section D proved to be one of the most contentious for the experts throughout the process. Di (i.e., The 
course architecture permits the online instructor to add content, activities and assessments to extend 
learning opportunities), D2 (i.e., The course accommodates multiple school calendars, e.g., block, 4x4 and 
traditional schedules), D6 (i.e., Prerequisite skills in the use of technology are identified), D7 (i.e., The 
course uses content-specific tools and software appropriately), D8 (i.e., The course is designed to meet 
internationally recognized interoperability standards), and D9 (i.e., Copyright and licensing status, 
including permission to share where applicable, is clearly stated and easily found) were all flagged for 
further discussion coming out of round one. There was concern that Di was not appropriate for all 
instructors, so adding “where applicable” at the end of the element was suggested by an expert. Both 
groups unanimously agreed with the revision. D9 was quickly agreed upon as well, with both groups 
acknowledging the importance of copyright laws. 
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D2 was a lengthier discussion for both groups. All experts agreed that giving calendar examples limited 
what an online course could fit into. Group B suggested that the element was not needed at, and voted to 
delete the element. Group A, on the other hand, simplified the wording and related it to the module 
design previously mentioned in Section B. 

In round two, the experts were split on how to handle both D6 and D7. For both groups, the conversation 
began with a possible combination the elements, using the suggestion “Prerequisite skills, course tools, 
and course software are identified and appropriate in relation to the students and course.” Louise 
mentioned to group A that the notion of prerequisite skills should be part of communication, but was not 
a function of course design. The other experts agreed, and removed “prerequisite skills” from the revision. 
Group B was fairly adamant that D6 had to remain in some form or another. As Amanda put it 

I’m looking at this from trying to explain to a parent, you know, why their student shouldn’t take 
this specific course because maybe they don’t meet those prereqs....prerequisite skills in the use of 
technology are identified. That is, that is something that they need to know how to do. How to 
navigate, you know, different parts of the course. And it might be course specific, meaning 
different courses will have different prereqs, but I don’t think you can delete this. 

After a bit more discussion, Group B approved the combination of D6 and D7 as suggested. 

From the round one and two comments, it appeared that some experts were not familiar with what D8 
(i.e., The course is designed to meet internationally recognized interoperability standards) was referring 
to. Even after further explanation, Group A was quick to delete the element, not viewing it as a necessary 
part of design. Group B took a different stance, viewing D8 as something that will be important in the 
future of design. Jason brought up that as instructors and students move from one proprietary software to 
another, it is important they have the ability to keep communicating and creating. The other experts 
agreed, and opted to keep the element. 

As with the previous elements, a final version of the suggestions had to be obtained for D2, D6, D7, and 
D8. D2 (i.e., The course accommodates multiple school calendars, e.g., block, 4x4, and traditional 
schedules) appeared to have middling support from both research and the experts, so the decision was 
made to eliminate the element. The additional thought was that the modules in the course could be 
manipulated to fit any calendar, so there was not an overwhelming need to mention this as a design 
requirement. It would instead fall to the instructor and institution to make the course work for them. 
There was a strong argument for keeping D6 (i.e., Prerequisite skills in the use of technology are 
identified), and the suggested combination of D6 and D7 (i.e., The course uses content-specific tools and 
software appropriately) was used. Finally, Group B’s suggestion that D8 (i.e., The course is designed to 
meet internationally recognized interoperability standards) would be relevant in the future of design was 
enough to keep the element intact. 
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Table 6 

Section E: Course Evaluation and Support Elements 


Accessing course effectiveness 

Round one 
average 

Round two responses 

Element El: The course provider uses multiple ways of 
assessing course effectiveness. 

2-75 

N/A 

Element E2: The course is evaluated using a 
continuous improvement cycle for effectiveness and 
the findings used as a basis for improvement. 

2.875 

N/A 

Course updates 

Round one 
average 

Round two responses 

Element E3: The course is updated periodically to 
ensure that the content is current. 

2.875 

N/A 

Certification 

Round one 
average 

Round two responses 

Element E4: Course instructors, whether face-to-face 
or virtual, are certificated and “highly qualified.” The 
online course teacher possesses a teaching credential 
from a state-licensing agency and is “highly-qualified” 
as defined under ESEA 

2-375 

Keep standard as is: 3 

Revise standard: 3 

Combine with another: 0 

Delete standard: 2 

Instructor and student support 

Round one 
average 

Round two responses 

Element E5: Professional development about the 
online course delivery system is offered by the 
provider to assure effective use of the courseware and 
various instructional media available. 

2.625 

N/A 

Element E6: The course provider offers technical 
support and course management assistance to 
students, the course instructor, and the school 
coordinator. 

2.325 

Keep standard as is: 4 

Revise standard: 2 

Combine with another: 0 

Delete standard: 2 

Element E7: Course instructors, whether face-to-face 
or virtual, have been provided professional 
development in the behavior, social and when 
necessary, emotional aspects of the learning 
environment. 

2.125 

Keep standard as is: 2 

Revise standard: 2 

Combine with another: 0 

Delete standard: 4 

Element E8: Course instructors, whether face-to-face 
or virtual, receive instructor professional 
development, which includes the support and use of a 
variety of communication modes to stimulate student 
engagement online. 

2.25 

Keep standard as is: 3 

Revise standard: 2 

Combine with another: 2 

Delete standard: 2 
(Revise/Combine: 1) 

Element E9: The provider assures that course 
instructors, whether face-to-face or virtual, are 
provided support, as needed, to ensure their 
effectiveness and success in meeting the needs of 
online students. 

2-75 

N/A 

Element E10: Students are offered an orientation to 
taking an online course before starting the 

2.25 

Keep standard as is: 5 

Revise standard: 1 
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coursework. 


Combine with another: 0 



Delete standard: 2 


Much like the previous section, E brought about much discussion as to how the elements pertained to 
course design, or if they did at all. E4 (i.e., Course instructors, whether face-to-face or virtual, are 
certificated and “highly qualified.” The online course teacher possesses a teaching credential from a state¬ 
licensing agency and is “highly-qualified” as defined under ESEA), E6 (i.e., The course provider offers 
technical support and course management assistance to students, the course instructor, and the school 
coordinator), E7 (i.e., Course instructors, whether face-to-face or virtual, have been provided professional 
development in the behavior, social, and when necessary, emotional aspects of the learning environment), 
E8 (i.e., Course instructors, whether face-to-face or virtual, receive instructor professional development, 
which includes the support and use of a variety of communication modes to stimulate student 
engagement online), and E10 (i.e., Students are offered an orientation to taking an online course before 
starting the coursework) were all forced into the discussion for round two. E2 (i.e., The course is evaluated 
using a continuous improvement cycle for effectiveness and the findings used as a basis for improvement) 
and E3 (i.e., The course is updated periodically to ensure that the content is current) were suggested to be 
combined by an expert in round one, and E4 through E10 were all put up for deletion in various round 
one suggestions. The conversation in round three began with combining E2 and E3. Group A believed the 
standards did not fit into design and should therefore be eliminated. Group B saw it differently, believing 
that the findings from the evaluation should be used to improve and update the course. However, there 
was concern over the use of periodically and what that actually meant. Kim suggested adding “as needed” 
to the end of the combined revision, and the rest of Group B agreed. 

During round one, it was suggested that elements E4 through E10 could be completely eliminated, as they 
do not relate to course design. Group A quickly moved to eliminate all remaining elements, with the 
exception of E6 (i.e., The course provider offers technical support and course management assistance to 
students, the course instructor, and the school coordinator). Louise was adamant that support should be 
built directly into the course, and not remain at the institution level 

It’s the program that is delivering, designing, and then delivering this online course that makes 
the determination how the technical assistance is going to be provided. But the given is that 
within that course design, is the tool for technical assistance. But it’s a programmatic decision. We 
don’t care...who provides the assistance. As long as it can be found. 

The rest of Group A agreed, and a revised version of E6 remained. Group B, however, came to the 
conclusion that E6, as well as the rest of the elements in the suggestion, could be eliminated. Kelly 
summed up the collective thought 

I think too there’s a lot of them that are...higher level program. The course provider in terms of 
technical support, they’re going to provide that. I see a lot of program level, like orientation for 
students, I think that’s program related piece, too. That should be for all students taking any 
online course within the program or whatever it might be. 
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Group B concurred, and elements E4 through E10 were deleted. 

Reviewing data and all reviewer comments, the suggested combining of E2 (i.e., The course is evaluated 
using a continuous improvement cycle for effectiveness and the findings used as a basis for improvement) 
and E3 (i.e., The course is updated periodically to ensure that the content is current) was accepted into the 
final rubric. Course design can be continuous and ongoing, meaning there should be an evaluation and 
improvement process in place. The revised version of E6 (i.e., The course provider offers technical 
support and course management assistance to students, the course instructor, and the school coordinator) 
was also added. Group A made a strong case for the need of technical support to be located in each course. 
While it does not matter who eventually supplies the support, there should be access to help for every 
instructor and student directly within the course. 

Table 7 


Section F & G: Suggested Elements and Revisions 


Suggested elements 

Round one 
average 

Round two responses 

Element Ft: The syllabus promotes a student plan of 
work with attainable expectations. 

2.625 

N/A 

Element F2: Technology is used to help increase self- 
efficacy of students. 

2.625 

N/A 

Element F3: Activities are designed to encourage 
students’ individual interests and goals. 

2.5 

N/A 

Element F4: The instructor understands student goals 
and personalizes support. 

2-75 

N/A 

Suggested revisions 

Round one 
average 

Round two responses 

Combine elements B4 and B6. 

2-75 

N/A 

Combine elements E6 and E7. 

2 

Keep revision as is: 1 

Revise revision: 1 

Combine with another: 0 

Delete revision: 5 


In round one, the experts were presented with four additional elements and two revisions. Elements Ft 
(i.e., The syllabus promotes a student plan of work with attainable expectations), F2 (i.e., Technology is 
used to help increase self-efficacy of students), and F3 (i.e., Activities are designed to encourage students’ 
individual interests and goals) were all readily accepted. In round two, there was a strong consensus to 
delete F4 (i.e., The instructor understands student goals and personalizes support). When mentioned in 
round three, there was no call for discussion from either group, and F4 was eliminated. Ft and F3 were 
placed in Section B under the Instructional Strategies and Activities subsection. F2 was located in Section 
D under the User Interface subsection. 

The revisions were split with the experts. There was strong support in round one to combine B4 (i.e., The 
course and course instructor provide students with multiple learning paths, based on student needs that 
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engage students in a variety of ways) and B6 (i.e., The course provides options for the instructor to adapt 
learning activities to accommodate students’ needs). The suggested revised wording was not brought up 
by experts for further discussion and was added to the final rubric. However, most experts did not believe 
E6 and E7 were closely related, and the combined suggestion was dropped. In the end, both elements 
were ultimately recommended for deletion by both expert groups. 


Discussion 

The first section of the iNACOL National Standards for Quality Online Courses (i.e., Section A: Content ) 
received a relatively high level of support from the experts during all three rounds of review. This result 
was not surprising, given the fact that these standards were primarily centered on structural and 
preparatory aspects of the online course. For example, in his study of course developers at a province¬ 
wide supplemental virtual school, Barbour (2005, 2007) reported several principles of effective course 
design that were focused on items like the consistency of navigation and structured course content. 
Similarly, students have also stressed the importance of structural and preparatoiy material in an online 
course. Gallini and Barron (2001-2002) reported that students preferred “a course structure with clear 
guidelines along with opportunities in the course to suggest alternative approaches to meeting course 
objectives” (p. 149), all aspects of structural and preparatory material found in an online course. Even 
most of the QM general standard areas (i.e., course overview and introduction, learning 
objectives/competencies, assessment and measurement, instructional materials, learner interaction and 
engagement, course technology, learner support, and accessibility) were focused on what online course 
designers would describe as structural and preparatoiy items (MarylandOnline, 2013). 

Considering the significant tie between instructional and course design, expert support for the majority of 
the Section B (i.e., Instructional Design) elements was not unexpected. There was agreement that 
opportunity for higher order thinking, differentiating, and active learning be taken into consideration 
when designing the course. This was also supported by Mastropieri et al. (2006), who discussed how 
differentiating helped middle school science students achieve higher score on both in-class unit and state 
exams. The largest obstacle in Section B was actually related to the wording of certain elements. Experts 
agreed that resource materials could help with mastery, as have been seen in the K-12 online learning 
literature with algebra students who used virtual manipulatives (Cavanaugh, 2013). The wording and 
redundant nature of certain elements led to combining parts of Section B. 

As a whole, the Section C (i.e., Student Assessment) elements were agreed upon and accepted by the 
experts in the revised rubric. This level of agreement was consistent with DiPietro (2010), who 
interviewed 16 online educators and found that participants agreed that assessment and feedback helped 
students engage with the content, along with meeting their individualized needs. In fact, as students 
become engaged with the learning, they are generally open to hearing feedback on how to improve and 
reach mastery of the subject material (Naidu, 2013). This feedback can be aided by the use of various 
resources, including rubrics, and by viewing course rubrics students become aware of expectations (Rice, 
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2012). As with Section B, the experts were mainly concerned with the wording of various elements, and 
moved forward with the section largely intact. 

Unlike the previous three areas, Section D (i.e., Technology) garnered more discussion with regards to 
both wording and how the elements pertained to course design. The experts agreed that flexibility was 
important to scheduling online courses, a notion that Wicks (2010) also supported. However, the experts 
thought an element specifically about different calendar types was unnecessary, and that element was 
subsequently deleted. Further, there was also open debate over interoperability of the course, with some 
experts not seeing the necessity of integration. However, Watson and Watson (2007) noted that LMSs 
needed to “truly become systemic, integrating systems seamlessly to allow for improved collaboration 
across systems among stakeholders” (p. 32). While many of the remaining elements were eventually 
reworded or combined, the experts were generally agreeable with the general sentiment found in Section 
D (i.e., that understanding that the technology used played an important part in course design). This is 
consistent with earlier studies into the design of K-12 online courses. For example, Barbour (2007) 
interviewed six online educators who found that minimal and simple navigation gave a consistency that 
was appreciated by the students. The same group of educators, however, didn’t feel that a course should 
shy away from multimedia and interactive elements, which could be used to enhance the curriculum. 

Section E (i.e., Course Evaluation and Support Elements) was by far the most retooled area from the 
experts, but this is not to say that the elements were not important when creating an online course. As a 
few of the experts pointed out, all the elements were significant, but simply belong to different rubrics, as 
opposed to one focused on online course design. For example, the VHS required a 26-week class in how to 
design a course that utilized the LMS (Zucker & Kozma, 2003). Further, the Illinois Virtual High School 
(IVHS) used a similar practice shell as well, as was pointed out by Barbour, Kinsella, Wicks, & Toker 
(2009). IVHS also realized that continuous support was needed, and offered monthly professional 
development using face-to-face, synchronous, or asynchronous methods. However, that did not mean that 
the need for professional development and support should be included in a rubric designed to measure 
quality online course design. Yet, not all elements from Section E were deleted. For example, continuous 
course updates were fully supported by experts, and was a practice utilized by many K-12 online learning 
programs (Ebert & Powell, 2015). 

The final areas were Section F and G (i.e., Suggested Elements and Revisions), which focused on the 
elements that were suggested for addition or elements that should be revised. The suggested elements 
looked to include student motivation in the course design structure, which was not a part of the original 
iNACOL standards. Three of the four suggestions were strongly supported by the experts, and found their 
way into the revised rubric without revisions. Both Chen and Jang (2010) and Kim, Park, and Cozart 
(2014) reported that motivation was an essential part of education, particularly in the online learning 
environment. As such, it was important that components that fostered student satisfaction in autonomy 
and self-efficacy were planned for within the online course design. 
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Conclusions and Implications 

After examining the iNACOL National Standards for Quality Online Courses (2011) based on current 
literature and research in phase one of the study (see Adelstein & Barbour, 2016), a revised set of 
standards were reviewed by eight experts for phase two. The review occurred over three rounds, with the 
first two happening via e-mail and the third through a video conference. During all three rounds the 
experts recommended that certain elements be kept, combined, or deleted using their knowledge and 
understanding of online course design as a guide. The end result was a K-12 online course design rubric 
based off the original iNACOL quality standards that was further revised and refined. 

The iNACOL standards, while praised by the experts, are purposefully broad, covering all aspects of online 
courses. The results of phase two of this study helped to bring essential online course design standards 
into focus. This specialized and more focused view may be able to help curtail how overwhelming the 
standards can appear, especially for those new to the field of K-12 online learning and designing online 
courses for a K-12 population. The revised rubric will allow stakeholders, including educators, course 
designers, and administrators, to focus specifically on the aspects of online course design, creating a 
stronger base upon which to build asynchronous online course content. 

Having said that, the researchers would recommend that further expert review be conducted. Due to time 
and resource constraints, the experts for this study were limited to eight individuals. Also, while the first 
two rounds were vital to giving the experts some guidance, the majority of the discussion related to and 
refinement of the individual elements occurred during the video conference; which was limited to 
approximately one hour. It would be beneficial to provide the experts multiple opportunities to video 
conference over the course of the refinement of the standards. Finally, the iNACOL standards were chosen 
due to their open, non-proprietary nature. However, there are also other widely used standards that could 
be used or supplemented as the basis for this model of expert discussion. As for our own line of inquiiy, 
with the expert review completed, the next phase of this particular study will test the application of the 
rubric. Three to five teams of two reviewers will apply the rubric against current K-12 online courses. 
Using inter-rater reliability, the researchers will examine the reliability and validity of the rubric. 
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Appendix 


Revised Rubric 


SECTION A: CONTENT 

Element 

Further Explanation 

Subsection: Academic Content Standards and Assessments 

Al: The course content and assignments are 
aligned with the state’s content standards, 
common core curriculum, or other accepted 
content standards set for Advanced Placement 
courses, technology, computer science, or other 
courses whose content is not included in the 
state standards. 

The content and assignments for the core 
courses are explicitly and thoroughly aligned to 
the credit granting state’s academic standards, 
curriculum frameworks and assessments. 
Advanced Placement® courses must be 
approved with the College Board and other 
elective courses should be aligned to other 
nationally accepted content standards such as 
computer science, technology courses, etc. 

Al Rating (1 = not applied, 2 = partially applied, 3 = applied): 



A2: The course content and assignments are of 
sufficient rigor, depth and breadth to teach the 
standards being addressed. 

The course componen 
instructional strategic 
and technology) are si 
rigorous such that sue 
the knowledge and ski 
standards upon compl 

ts (obje 
s, conte 
ifficient 
cessful 
11s requ 
etion of 

dives, assessments, 
nt, assignments 
ly broad, deep and 
students will have 
ired by the 
the course. 

A2 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



A3: All course materials are available to 
students at course start. 

Before the course begi 
learning resources tha 
online course. These c 
instructional material 
and other software, wl 

ns, stud 
t are ut 
ould inc 
5 links t 
lich stu 

ents are provided 
ilized during the 
Iude textbooks, 
o browser plugins, 
dents must install. 

A3 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



Subsection: Course Overview and Introduction 



A4: A complete course overview and syllabus, 
which clearly states course goals and objectives, 
are included. Course goals are consistent with 
course requirements and are measurable in 
multiple ways. 

Within the learning management system the 
syllabus and overview objectives are present, 
explicitly stated, and can be easily found by 
students. The syllabus and overview objectives 
include: course objectives and student learning 
outcomes; assignments; student expectations; 
time requirements; required materials; the 
grading policy; teacher-student, teacher-parent 
contact policies; the intended audience; and the 
content scope and sequence. 
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A4 Rating (1 = not applied, 2 = partially applied, 3 = applied): 


SECTION A: CONTENT 

Element 

Further Explanation 

A5: Information is provided to students, parents 
and mentors on how to communicate with the 
online instructor and course provider. 

Instructor information is provided to students 
with contact, availability, and biographical 
information. Information on how to contact the 
instructor via phone, email, and/or online 
messaging tools is provided within the contact 
information. If regular contact with the 
instructor is required as part of the course, clear 
expectations for meeting this requirement are 
posted within the course. 

A5 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



Subsection: Legal and Acceptable Use Policies 



A6: The course reflects multi-cultural 
education, and the content is accurate, current 
and free of bias or advertising. 

The course creates equal educational 
opportunities for students from diverse racial, 
ethnic, social-class and cultural groups. The 
content is up to date, accurate and free of any 
bias. 

A6 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



A7: Expectations for academic integrity, use of 
copyrighted materials, plagiarism and 
netiquette (Internet etiquette) regarding lesson 
activities, discussions, and e-mail 
communications are clearly stated. 

A “Code of Conduct” 
standards, copyright 
expectations is providi 

includii 
ind aca< 
ed. 

lg netiquette 
iemic integrity 

A7 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



A8: Privacy policies are clearly stated. 

A policy statement is j 
provider’s website ant 
management system d 
organization’s inform 
dissemination practice 

>osted o 
I/or in t 
isclosin 
ation ga 
s. 

n the course 
he learning 
g the 

thering and 

A8 Rating (1 = not applied, 2 = partially applied, 3 = applied): 
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SECTION A: CONTENT 

Element 

Further Explanation 

Instructor Resources 

A9: Online instructor resources (e.g. 
assessment, assignment answers and 
explanations, notes) are included. Pedagogy 
behind the resources are shared with 
instructors. 

Resources and notes, including assessments and 
access to answers, explanations to aid online 
instructors in teaching and facilitating the 
course are included within the learning 
management system. 

A9 Rating (1 = not applied, 2 = partially applied, 3 = applied): 
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SECTION B: INSTRUCTIONAL DESIGN ELEMENTS 

Element 

Further Explanation 

Subsection: Instructional and Audience Analysis 

Bl: Course design reflects a clear 
understanding of all students’ needs and 
incorporates varied ways to learn and master 
the curriculum. 

A variety of instructional and assessment 
methods, materials and assessments are 
used throughout the course, which allow 
students to demonstrate their achievement 
of the goals and objectives of the course. 

Bl Rating (1 = not applied, 2 = partially applied, 3 = applied): 



Subsection: Modules and Resources 



B2: The course is organized by modules. Course 
design provides students with resources (e.g. 
alternate assignments, multimedia, simulations) 
that enrich course content. Each module 
includes an overview of the key objectives that 
incorporate a variety of activities, assignments, 
and resources to provide multiple learning 
opportunities for students to master the content. 

The course is organized by modules that fall 
into a logical sequence. At the start of each 
module, an overview is posted describing 
the activities, assignments, assessments, and 
resources to be used to complete the key 
objectives. A variety of activities, 
assignments, assessments, and resources are 
used to provide students with different 
paths to master the content. A wide variety 
of supplemental tools are clearly identified 
and readily available as well. 

B2 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



Subsection: Instructional Strategies and Activities 



B3: The course instruction includes activities 
that engage students in active learning. 

The course provides multiple opportunities 
for students to be actively engaged in the 
content that includes meaningful and 
authentic learning experiences such as 
collaborative learning groups, student-led 
review sessions, games, analysis or reactions 
to videos, discussions, concept mapping, 
analyzing case studies, etc. 

B3 Rating (1 = not applied, 2 = partially applied, 3 = applied): 
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SECTION B: INSTRUCTIONAL DESIGN ELEMENTS 

Element 

Further Explanation 

B4: The course provides options for instructors 
to adapt learning activities based on student 
needs, allowing for the course and instructors to 
offer learning paths that engage in a variety of 
ways. 

Students are given a variety of activities, 
assignments, assessments and resources to 
allow them to successfully master the 
content. If a student is unsuccessful with 
mastering a particular concept or is not 
challenged with the current module, the 
course content provides the instructor with 
suggestions they are able to use in order to 
provide additional remediation activities or 
alternative assignments. The instructor has 
access to adapt the course to meet the 
students’ needs by providing additional 
assignments, resources and activities for 
remediation or enrichments for the course. 

B4 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



B5: The course provides opportunities for 
students to engage in higher-order thinking, 
critical reasoning activities and thinking in 
increasingly complex ways. 

Assignments, activit 
provide opportunity 
their thinking beyon 
comprehension into 
situations, synthesizi 
evaluating an argurr 
include open-ended 
encourage students 
classify information, 
group work, decisioi 
patterns should also 
course activities. 

ies and 
;s for si 
d knov 
the rea 
ing infc 
lent. A< 
questio 
o catej] 
Oppoi 
i-maki 
be incl 

assessments 
tudents to elevate 
pledge and 
lm of analyzing 
irmation or 

Aivities should 
ns and 
;orize and 
rtunities for 
ng and finding 
uded in the 

B5 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



B6: Readability levels, written language 
assignments and mathematical requirements 
are appropriate for the course content and 
grade-level expectations. 

The course content s 
appropriate readabi 
level of the student 
level should be proir 
within the course de 

hould 
lity lev 
udienc 
linenth 
scriptk 

be written at 
els for the grade 
e and the grade 
/ explained 
>n. 

B6 Rating (1 = not applied, 2 = partially applied, 3 = applied): 
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SECTION B: INSTRUCTIONAL DESIGN ELEMENTS 

Element 

Further Explanation 

B7: The syllabus promotes a student plan of 
work with attainable expectations. 

The syllabus provides an academic outline for 
students in the course, which includes academic 
expectations at specific intervals. 

B7 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



B8: Activities are designed to encourage 
students’ individual interests and goals. 

The course provides a 
which are broad enou 
connections. The conn 
such as personal inter 

ctivities 
gh to all 
ections 
ests, goi 

and assignments 
ow for student 
are real world, 
ds, or situations. 

B8 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



Subsection: Communication and Interaction 



B9: The course design provides opportunities 
for appropriate instructor-student interaction, 
including opportunities for timely and frequent 
feedback about student progress. 

Learning activities and other opportunities 
are created to foster instructor-student 
interaction. Students receive timely and 
frequent feedback on their progress that 
emphasizes the intended learner outcomes. 
The feedback is highly individualized, 
detailed, and recommends specific, 
individualized improvement, and strategies 
to encourage continued progress toward 
mastery. 

B9 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



BIO: The course design includes explicit 
communication/activities/tools at multiple 
intervals throughout the course. The instructor 
confirms whether students are engaged and are 
progressing through the course. The instructor 
will follow program guidelines to address non- 
responsive students. 

Instructor-student ii 
enough in the courst 
participation by all 
throughout the cour 

iteracti 

to con 

student 

se. 

ions begin early 
firm active 
s and continue 

BIO Rating (1 = not applied, 2 = partially applied, 3 = applied): 
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SECTION B: INSTRUCTIONAL DESIGN ELEMENTS 

Element 

Further Explanation 

Bll: The course provides opportunities (e.g. 
student-instructor, student-student interaction, 
student-course content, student-LMS) for 
mastery and application of the material. 

Learning activities and other learning 
opportunities are developed to foster 
student-instructor, student-student, and 
student-LMS interaction. The technology 
and course content encourage exchanges 
amongst the instructor and students 
through email, discussions, synchronous 
chats, simulations, lab activities and other 
group projects. Within the grading policy, 
guidelines defining student participation 
and expectations are provided. 

Threaded and/or synchronous discussions 
are available for developing community, 
asking and finding answers to questions 
about the course, and around the content. 
Access is available to groups or individual 
students based on the purpose of the 
activity. Rules, roles, and expectations for 
the discussion are clear and posted within 
the discussion forum. 

Bll Rating (1 = not applied, 2 = partially applied, 3 = applied): 
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SECTION C: STUDENT ASSESSMENT ELEMENTS 

Element 

Further Explanation 

Subsection: Evaluation Strategies 

Cl: Student evaluation strategies are consistent 
with course goals and objectives, are 
representative of the scope of the course and are 
clearly stated. 

The strategies used to assess students 
throughout the course are consistent with and 
aligned to what is presented in the course goals 
and objectives document posted within the 
course. 

Cl Rating (1 = not applied, 2 = partially applied, 3 = applied): 



C2: The course structure includes adequate and 
appropriate methods and procedures to assess 
students’ mastery of content. 

Assessment types are 
knowledge being teste 
assessments (that infoi 
and summative assess] 
mastery) are a part of 
Student-selected asses 
learners to demonstra 
ways, are available. 

matchec 
d. Both 
rm and 
ments (f 
the cou 

sment o 

te mash 

1 to the level of 
formative 
support learning) 
hat demonstrate 
rse structure, 
ptions, enabling 
;ry in different 

C2 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



Subsection: Feedback 



C3: Ongoing and varied quality assessments 
aligned with course learning outcomes are 
conducted throughout the course to guide 
student instruction. 

The course provides quality and ongoing 
formative assessments to check for student 
understanding and to ensure they are prepared 
for the next lesson. Initial pre-tests may be 
provided to assess student readiness. 

C3 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



C4: Assessment strategies and tools make the 
student continuously aware of his/her progress 
in class and mastery of the content. 

Feedback tools and pr 
course to allow studen 
monitor their academi 

ocedurt 
ts to pe 
c progr 

;s are built into the 
riodically self- 

ess. 

C4 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



Subsection: Assessment Resources and Materials 



C5: Assessment materials provide the instructor 
with the flexibility to assess students in a variety 
of ways. 

Multiple versions of tests, test banks and other 
resources that support alternative evaluation 
methods are available. 

C5 Rating (1 = not applied, 2 = partially applied, 3 = applied): 
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SECTION C: STUDENT ASSESSMENT ELEMENTS 

Element 

Further Explanation 

C6: Suggested grading rubrics are provided to 
the instructor. The instructor will share a 
chosen grading rubric with students. 

Rubrics, rationale, and/or characteristics are 
provided for each graded assignment. The 
instructor will make the final selection, which 
will then be shared with the students. 

C6 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



C7: The grading policy and practices are easy to 
understand and clearly communicated to 
students and parents. 

Grading policies and i 
and clearly defined an 
penalties that may be 
extra credit opportuni 

iractice 
d may i 

assessec 

ties. 

s are easy to read 
nclude any 
to grades and/or 

C7 Rating (1 = not applied, 2 = partially applied, 3 = applied): 
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SECTION D: TECHNOLOGY 

Element 

Further Explanation 

Subsection: Course Architecture 

Dl: The course architecture permits the online 
instructor to add content, activities and 
assessments to extend learning opportunities 
where applicable. 

The instructor of record for the course has 
access to make additions to the content within 
the learning management system (LMS). Access 
should allow the instructor to add content, 
activities, and assessments, where appropriate. 
The content from the “original” base course is 
left unchanged. 

Dl Rating (1 = not applied, 2 = partially applied, 3 = applied): 



Subsection: User Interface 



D2: Clear and consistent navigation is present 
throughout the course. 

The course utilizes consistent and predictable 
navigation methods. Students can move logically 
and easily between areas of the course; color, 
graphics and icons are used to guide the student 
through the course; and a consistent look and 
feel exist throughout the course (consistent text, 
colors, bullets, and heading styles). Minimal 
training is required to navigate the course. 

D2 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



D3: Rich media are provided in multiple 
formats for ease of use and access in order to 
address diverse student needs. 

Course makes maximi 
capabilities of the onli 
these resources availal 
(video, CDs, podcasts) 

im use 

ne medi 
ble by a 

of the robust 
um and makes 
lternative means 

D3 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



D4: Technology is used to help increase self- 
efficacy of students. 

Technology used in th 
the student’s ability to 
goals set forth by the s 

e course 

accom 

yllabus 

does not hinder 
jlish the academic 

D4 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



Subsection: Technology Requirements and Interoperability 



D5: All technology requirements (including 
hardware, browser, software, etc.) are specified. 

All technology requirements (including 
hardware, browser, software, etc.) are identified 
in the course description or during the student 
registration process and specified to students 
before they begin the course. 

D5 Rating (1 = not applied, 2 = partially applied, 3 = applied): 
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SECTION D: TECHNOLOGY 

Element 

Further Explanation 

D6: Prerequisite skills, course tools, and course 
software are identified and appropriate in 
relation to the students and course. 

All prerequisite technology skills, software, and 
online tools necessary for the specific class are 
identified in the course description or during the 
registration process and are shared with 
students before they begin the course. Tools 
should be appropriate, necessary for teaching 
and/or enriching the lesson, cross-platform and 
free to the student (or built into the course). 

D6 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



D7: The course is designed to meet 
internationally recognized interoperability 
standards. 

Interoperability techn 
sharing content amon; 
management systems 
questions, assessments 

ical stai 
’ differ* 
and ens 
and re 

idards allow 
;nt learning 
ure sharing of 
suits with others. 

D7 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



D8: Copyright and licensing status, including 
permission to share where applicable, is clearly 
stated and easily found. 

Course developers or 
the copyright and licei 
including permission t 
Copyright and licensii 
readily available, und< 
standardized in terms 

publish 
rising st 
o share 
lg infor 
jrstand 
of use. 

ers clearly state 
atus of all content, 
where applicable, 
mation should be 
able and 

D8 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



Subsection: Accessibility 



D9: Course materials and activities are designed 
to provide appropriate access to all students. 

The course, developed with universal design 
principles in mind, conforms to the U.S. Section 
504 and Section 508 provisions for electronic 
and information technology as well as the 

W3C’s Web Content Accessibility Guidelines 
(WCAG 2.0). 

Through the use of web accessibility evaluation 
tools, all web pages required for students to 
engage in online education (e.g., registration, 
library, course materials, grade retrieval) are 
validated to conform to accessibility standards. 
NIMAS is used to ensure textbooks and other 
instructional materials are accessible to the 
visually impaired. 

D9 Rating (1 = not applied, 2 = partially applied, 3 = applied): 
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SECTION D: TECHNOLOGY 

Element 

Further Explanation 

Subsection: Resources and Materials 

DIO: Student information remains confidential, 
as required by the Family Educational Rights 
and Privacy Act (FERPA). 

Defined course procedures for reporting grade 
and student information complies with the 

Family Educational Rights and Privacy Act 
(FERPA) 

http://www.ed.gov/policy/gen/guid/fpco/ferpa/ 
index.html) posted within the course. 

DIO Rating (1 = not applied, 2 = partially applied, 3 = applied): 
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SECTION E: COURSE EVALUATION AND SUPPORT ELEMENTS 

Element 

Further Explanation 

Subsection: Accessing Course Effectiveness 

El: The course provider uses multiple ways of 
assessing course effectiveness. 

A combination of student, instructor, content 
experts, instructional designer and outside 
reviewers may be used to evaluate the course for 
effectiveness. A variety of methods may be used 
including course evaluations, student 
completion rates, satisfaction surveys, peer 
review, teacher and student feedback, and 
student performance on in-course as well as 
state or national assessments. University 
researchers have been encouraged to conduct 
studies on the effectiveness of the course. 

El Rating (1 = not applied, 2 = partially applied, 3 = applied): 



Subsection: Course Updates 



E2: The course is evaluated using a continuous 
improvement cycle for effectiveness. The 
findings are used to improve and update the 
course content as needed. 

The provider indicates the frequency of course 
evaluations, whether reviews are conducted 
internally or externally, and how the provider 
uses evaluation results to improve courses. 
Courses should be reviewed to keep the content 
current, engaging, and relevant. 

E2 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



Subsection: Instructor and Student Support 



E3: Technical support and course management 
assistance are provided to students, the course 
instructor, and the school coordinator. 

Online technical help and support should be 
available any time. If 24/7 support is not 
available, support hours are clearly posted 
within the course or on the online program’s 
website and a maximum response time is noted. 
Assistance may take the form of Frequently 
Asked Questions, training resources, mentors, 
or peer support. 

E3 Rating (1 = not applied, 2 = partially applied, 3 = applied): 
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